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ABSTRACT 
The evaluations made by 36 Navy fighter pilots of 64 air-to-air 
combat situations are statistically analyzed to detect any significant 
differences between pilots' evaluative techniques in relation to their 
rank, flight hours and combat missions. Predictor equations are computed 
and used ina self-analyzing, self-modifying artificial intelligence 


program modeled on an instructor-flight student interactive situation. 
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E. INTRODUCTION 


From the legendary Baron von Richtofen to the popular Snoopy 
flying his doghouse through the comic strips, the image of the fighter 
pilot is one of an enigma capable of performing complex and dangerous 
tasks in an environment where fractions of a second can spell the dif- 
ference between success and filure, life and death. This fascinating 
picture of a highly trained, assured and swaggering superbeing beckons 
to be studied; to have the mystique lifted. Hence, the impetus of this 
research: the fighter pilot. 

Other than the satisfaction of human curiosity, the study of the 
fighter pilot -in this case the Naval fighter pilot- can ultimately result 
in improved training and selection of future pilots by understanding the 
activities of successful pilots; improved combat techniques by adaptation 
of the successful pilots' techniques: improved surface-to-air and air-to- 
air communications through better understanding; and improved electronic 


and mechanical pilot aids to enhance pilot performance. 


À. OPJEC TIVES 
The objectives of this research lay in two major areas. The first 
was to address the questions of how Navy fighter pilots evaluate an air- 


to-air combat threat, i.e., pilot versus pilot situations. How does the 


pilot reach his decision? Can this decision be somehow quantified for 


computer applications? What differences exist among pilots in their 
evaluations of. air-to-air threat? 

The second major area lies in the field of artificial intelligence. 
Classically concerned with games such as chess and checkers with well 
defined rules for determining success and failure [1, 2], artificial intel- 
ligence has seen little application in real life "games." A notable 
exception is Clarkson's portfolio program which has reached partial 
Are enent will suecesstul bank trust administrator [3]. Heuristics, 
such as the minimax technique whereby a player will always attempt to 
minimize his maximum loss [4], and binary choice selection [5], although 
fine for board games, find little application in games of life. As stated 
by Rigney [6]: 

NEO decision tasks in which the outcome is of great consequence 

and the time for decision is short, as in many military, business, 

and medical decisions, the strategy the decision maker uses is 

Of special interest.” 

From the strategies hopefully revealed in the attainment of the first 
objective, an artificial intelligence application is sought. As brought 
out by Shepard [7], an obvious disparity exists between "the effortless- 
ness and surety of most perceptual decisions and the painful hesitation 
and doubt characteristic of these 'higher level’ decisions" and strategies. 


To Overcome this disparity ina "game" of ill defined rules and high stakes 


is the objective of the artificial intelligence application. 





B DEFINITIONS 

At this time some definitions of terms used throughout this research 
should be brought forward. Artificial intelligence as used in this study 
encompasses the construction of "computer programs which exhibit 
behavior that we call 'intelligent behavior' when we observe it in human 
beings [8]." 

The area of artificial intelligence most apt for this study is 
decision making in.game playing. Hence the game is defined as air- 


to-air combat as experienced in the Vietnam area of operations. 


C. ASSUMPTIONS 
Certain assumptions used throughout the study are: 


(1) Data such as military rank and threat evaluation decisions 
can be placed on a quantifiable continuum. 


(2) Since no criteria of deciding success or failure in the game 
exist short of actual combat results, expertise in the game 
will decide successes or losses. 


(3) Expertise is directly proportional to military rank, flight 
hours and total number of combat missions. 


(4) The data base used is a representative sample of Navy 
fighter pilots. 





I Ir Ip EDU RESDETER MINING DECISION STRATEGY 


In an attempt to isolate and identify the important variables con- 


sidered by a fighter pilot in his evaluation of an air~to-air combat 


situation, a mail-out survey was designed. In the survey the subjects 


were asked to assume the following hypothetical situation: 


(1) 


(2) 


(3) 


(4) 


The subject is an airborne Target Area Combat Air Patrol 
(TARCAP) over enemy territory in Southeast Asia. 


The subject's aircraft is configured and armed for air-to-air 
combat only; hence, a fighter. 


The subject's primary mission is to intercept and destroy 
enemy air raids threatening friendly airborne strike forces. 


The TARCAP and immediate strike group is the only friendly 
force in the area. Therefore, the subject is free to fire at 
any threat without positive visual identification. 


Eleven different items of additional information were then presented 


to the subject. These were presented in pairs yielding a total of 55 


Kanopi í ions The items of information were: 


(1) 


(2) 
(3) 
(1) 
(5) 
(6) 
(7) 


Relative position of enemy aircraft from the friendly strike 
group: 


Type of enemy aircraft. 

Number of enemy aircraft. 

Number of friendly aircraft. 

Range from friendly aircraft to enemy aircraft. 
Speed of enemy aircraft. 


Friendly aircraft capabilities. 
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(8) Altitude of enemy forces. 
(9) Anti-aircraft activity in the area. 
(10) Bearing of enemy from friendly forces. 
(UA Closure rate of friendly and enemy forces. 

The subject then had to choose which one of the pieces of infor- 
mation presented in each pair would be of greater value in evaluating 
the threat of, and in planning engagement tactics against, enemy air 
units. An additional alternative was given whereby each member of the 
pair could be evaluated as being equal in importance. Appendix A con- 
tains a sample of this questionnaire. Ninety-seven Naval aviators 
responded to this questionnaire of which 68 or 70.1% were pilots and 
29 or 29.9% were Radar Intercept Officers, the second crewman in the 
F-4 Phantom jet. Eleven or 11.3% were F-8 Crusader pilots, the rest 
of the pilots being F-4 pilots. Sixty-three or 65.0% of the aviators had 
combat experience over Korea or Vietnam. Nine or 9.4% had engaged in 
actual air-to-air combat. 

From a scaling of the variables from the pair-comparisons data 
the variables were ranked as follows (in descending order of importance): 

(1) Relative position of enemy aircraft. 

(2) Bearing of enemy. 

(3) Range of enemy. 

(4) Altitude of enemy. 


(5) Friendly aircraft capabilities. 
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(G) Number of enemy. 

(7) Rate of closure. 

(8) Anti~aircraft activity. 

(9) Type of enemy aircraft. 

(10) Speed of enemy. 

(11) Number of friendly. 

From the eue of the first survey, and with the consideration 
of the type of information available to the pilot in the cockpit, a second 
mail-out survey was designed. The variables considered in this survey 
were limited to six in number. Fuel states and rules of engagement were 
added for realism. The remaining four variables were considered as being 
information a pilot would have at his access from either his cockpit 
instruments or via a surface or airborne controller. The variables were 
limited to two states, hence 25 or 64 possible combinations were con- 
sidered by each subject. The variables and their states were: 

(D Fuel above bingo; the maximum amount of fuel that can be 
utilized before fuel level drops below that necessary for a 
safe return to home base. 

(a) 1000 pounds. 
(b) 2 SUN pounds: 


(2) Rules of engagement; when the subject may take the suspected 
aircraft under fire. 


(a) Eyeball (positive visual identification is required). 


(b) PASSES tee (positive visual identification is not 
required). 
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Enemy's relative bearing from the subject. 
(am SS 

(b) SS z 

Enemy's range from the subject. 

(a) Eyeball or within 4 nautical miles. 
(b) 20 nautical miles. 

Enemy's true heading or direction of flight. 
(a) nes 

(b) 2 ON 


Number of enemy aircraft. 


The subjects were asked for their rank, total number of flight 


hours and total number of combat missions. The following background 
scenario was used for each of the 64 situations: 


1 
“You are the flight leader of a section of F-4's armed with two 
Sparrows2 and two Sidewinders.? Assume for this exercise that the 
aircraft's weapon systems are up in every respect. You are providing 


1 

A section normally consists of two F-4's. 
o M unes: 

Beam riding guided missile. 


3 
Heat seeking guided missile. 


l. 





TARCAP for a division of A-7's who have just completed a strike and are 
egressing from the target. area. You are feet-dry over North Vietnam 
(20 nm to the coast). The AA and SAM? defenses in the immediate area 
are light to moderate. You have limited GCI? facilities operating for 
you and the enemy has excellent ground radar control. 

The enemy aircraft are assessed to be MIG-21's at 15,000 feet and 
500 kts. You are 10,000 feet and 450 kts. heading for your carrier 
(360° relative). 


The weather in the area is clear and 15+ visibility. There are 
several flights of attack aircraft still feet-dry, exact position unknown. 


The MIG's have demonstrated an air-to-air missile capability." 

Figure 1 shows the way the tactical situations were presented to the 
subjects, analogous to a radar scope or maneuvering board's display. 

Two major evaluation questions were asked of each subject in con- 
junction with each of the 64 situations considered. These questions 
are shown in Figure 2. 

The questionnaire was sent to a number of Navy fighter squadrons 
on both the East and West Coasts. The aircraft in service at each of 
the squadrons was the F-4 Phantom. Thirty-six pilots responded. These 
pilots are described in Appendix B. Table 1 contains a description of 


the pilots by rank, hours and combat missions. The overall average 


Our land vice feet wet or over water. 
5 N 
Surface-to-Air Missile. 


ame Control Intercept. 
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Figure 1 


TACTICAL SITUATION PRESENTATION 
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Figure Z 


CI OI OO SED S II DACH TACTICAL DISPLAY 


In this tactical situation (check one answer): 


Le l'd have no choice; there would be an engagement. 

De I'd have a choice on whether to engage or not, and I'd 
engage. 

ce I wouldn't engage. 


Indicate what aircraft losses you would predict, if there were an 
engagement. 


Enemy Losses (no. of aircraft) 


Friendly Losses (mo. of aircraft) 
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respondee had a rank between Lieutenant and Lieutenant Commander 
(3.47)! , 1994.03 flight hours and 124.11 combat missions. Rank was 
assumed to be an attribute providing ordinal data. 

The data from the surveys were coded and placed on a magnetic 
disk for statistical analysis. The rules of engagement, being qualitative 
variables, were arbitrarily coded as a 0 for eyeball and 1 for missiles- 
free. A measure of central tendency (arithmetic mean) and measure of 
dispersion (standard deviation) was computed for each biographical 
variable (rank, flight hours, combat missions), each primary tactical 
decision and each secondary enemy and friendly kill decision made for 
the entire sample. Means and standard deviations of these variables 
and decisions were then computed for each subset of the sample defined 
by biographical data groupings, i.e., all Commanders, all Lieutenant 
Commanders, all Lieutenants, all Lieutenants (junior grade), all pilots 
with more than 4000 hours, all with less than 4000 and more than 3000, 
all with less than 3000 and more than 2000, all with less than 2000 and 
more than 1000, all with less than 1000, all pilots with over 300 missions, 
all with less than 300 and more than 200, all with less than 200 and 
more than 100, all with less than 100 and more than one mission and 
those pilots with no combat missions. Likewise, the mean and standard 


deviation of each decision was computed for the subset of the sample 


defined by holding each of the situational variables constant at one of 


Iu 
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its values and then the other while the remaining situational variables 
were free to assume either of their two values. 

In attempting to find a relationship between the biographical and 
situational variables and the decisions reached, correlation coefficients 
were computed between these variables and decisions. These correlation 
coefficients were computed for the entire sample, for each subset de- 
fined by the ey groupings, and for each respondee. These 
findings were presented to a panel of Naval aviators who were asked for 
their opinions and suggestions. These aviators are described in Appendix 
C. From their suggestions new variables made up of combinations of 
the original six variables were added to the data base and the same 
analysis again undertaken. This cycle was repeated until the panel con- 
sidered the results to be reasonable. 

In an attempt to distinguish any discernible differences between 
the decisions reached for each biographical subset within the rank, hour 
and mission groupings, a discriminant analysis was performed. In this 
analysis, a linear function for each subset within a group is computed 
using the subset characteristic (specific rank, hours grouping, or 
mission Grouping) as the dependent variable and the decisions reached 
as the independent variables. These functions are then evaluated for 
each case and the hypothesis that the subset means are the same is 


tested [9]. 


In an attempt to predict a pilot or group of pilots' decisions, a. 
sequence of multiple linear equations computed using stepwise multiple 
regression techniques were also produced for the entire sample and each 
subset. At each step in this regression procedure, one situation variable 
is added. The variable added is the one which makes the greatest re- 
duction in the error sum of squares (the variable with the highest partial 
correlation with the dependent variable at that step). The production 
stops when the error sum of squares decreases less than .001 or until 
all variables are exhausted [10]. The correlation coefficient between 
these equation predictions and the actual decision reached was then 
computed for each situation examined. These equations and correlations 


were then presented to the panel of aviators for their evaluation. 
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HI. RESULTS OF DATA ANALYSIS 


A. VARIABLES ADDED AND DELETED 

Since bearing and heading, although quantitative items Of antor= 
mation shown to be important in the first survey, did not represent con- 
tinuums that could be conveniently statistically manipulated, both 
were deleted in Javor Oi q new variable. The reason this was done is 
apparent when evaluating, for example, bearings E G0 We NE 
Although separated by only 2° on a compass rose, a quantity evaluated 
by the panel of aviators as negligible, the values are separated by 
357°, which is not negligible. Also, the two variables' effects on à 
pilot's evaluation appcared to be interactive. An enemy directly behind 
a friendly (bearing 180º) was Be es dstterently if the enemy Was 
pointed at the friendly (heading Seo or eolnting away (heading 160 8 
Ii s pointing information is not directly available to the pilot from either 
his cockpit displays Or GCI input. Yet the pilot does mentally integrate 
the bearing and heading information into pointing information. Conse- 
quently, a function of both bearing and heading was created. This 
function, called "danger angle, " is described by the following equation: 


Danger (bearing, heading) = (sin (bearing)/ | sin (bearing) | ) 
x [bearing - (heading * 180) modulus 360]. 


Similarly, although pilots considered range and fuel-remaining as 


important variables independently, the aviator panel believed that a 
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link existed between these two factors. For instance, a fighter pilot's 
behavior in a situation of long range with low fuel remaining would 
probably be different from that under a situation of short range and a 
large amount of fuel remaining. Again, this would be a case of pilot 
integration of a cockpit display (fuel remaining) and a GCI input or 
radar display (range) ue a range per fuel state factor. Consequently 
a range-fuel ratio (MPG) was created as described by the following 
equation: 

MPG (range, fuel remaining) = range/fuel remaining. 

As shown in the first mail-out questionnaire results, a relationship 
between the enemy’s speed and heading and friendly speed and heading, 
or closure rate is considered by fighter pilots (See EXPERIMENTAL 
PROCEDURE DETERMINING DECISION STRATEGY). Again, pilot inte- 
mation Ol Cockpit and GCI inputs appears to occur: the ae projecting 
the parallel enemy speed vector into his own. As a result, a speed 
differential function was created for the tactical scenario: 


Speed (heading, bearing) = 500 x cos (heading) - (cos (bearing)/ 
| cos (bearing) | ) x 450. 


In both the speed and danger equations, a value of +l was assigned to 
the ratio of the trigonometric function and its absolute value when that 


function was equal to zero. 
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Br CORRELATIONS 

The primary tactical decision made by the subjects was assumed 
to be ordinally quantifiable as indicated below: 

(1) Pilot has no choice, an engagement would occur. 

(2) Pilot has a choice and would elect to engage the enemy. 

(3) Pilot would not engage the enemy. 
The secondary decision was in two parts, number of enemy losses pre- 
dicted and number of friendly losses predicted if an engagement were 
tO OCCUT. 

The biographical data's correlation with the decision is shown in 


Table Ji tor all pilots. 


TABLE II 


CORRELATION OF BIOGRAPHICAL DATA WITH PRIMARY 
AND SECONDARY DECISIONS 


Tactical Enemy Friendly 

Decision Kills ‘ills 
Rank — 0690 .054] xod 
Hour -,0394 20528 1039 
Missions - .0333 SOS 1265 


The lack of correlations significantly different from zero is striking. 
Only with the relationship of rank to friendly kills predicted can a 
coefficient of correlation be found of any sizeable magnitude (+.2195). 
Discriminant function analysis revealed no significant differences 
between the pilots when they were categorized by rank, hour or mission 
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with respect to the decisions they made (at a significance level of 
< .05) (See Appendix D). The panel of aviators considered this (post 
hoc) to be a result of standardized training and operational squadron 
procedures. They also expected a positive correlation between rank 
and expected friendly kills as a result of more realistic evaluation 
techniques acquired with time spent in operational billets. A decline 
in youthful optimism as one ascends the rank and age structure was also 
expected to enhance this postulated positive correlation. Appendix F 
contains the means and standard deviation of the decisions reached by 
the biographical groupings. 

The situational variables’ correlations with the decisions are 


Shown.in Table III. 


TABLE IH 


CORRELATIONS OF SITUATIONAL VARIABLES WITH 
PRIMARY TACTICAL DECISIONS AND SECONDARY 
ERE po DNE MY AnD FRIENDLY KILL DECISIONS 


ee mee! Enemy Friendly 

Decision Kills Kills 
MPG „4259 -.2087 o2 
Speed 22.9 9:9 000 0552 
Danger .2864 Dg 20201 
Fuel Remaining -,1708 .3400 20208 
Range 4209 .0163 . 1/46 
Number of Enemy mS 0231 221 
Rules of Engagement -.0760 BAT 0.512 
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The panel of aviators, however, felt a chronological relationship 
existed between the decisions, that is, the pilot evaluated the situation 
as a whole and then reached his primary tactical decision. After that 
decision had been reached, and with that result kept in mind, it was 
thought that the pilot then made his secondary decisions. Suggestions 
by the author that the decisions of kill possibilities would precede and 
influence the tactical decision were strongly rejected by the panel as 
Mirealistic trom their own experiences. Table IV shows the correlation 


between the primary and secondary decisions. 


TABLE IV 


CORRELATION BETWEEN PRIMARY AND 
SECONDARY DECISIONS 


Enemy Kills Friendly Kills 
Tactical Decision -, 4247 -.2307 


The reader should note that the biographical variable of rank is 
the third most important variable when predicting secondary friendly 


mills. 


©. VARIABLE MANIPULATION 

The effects of holding specific situation variables constant on the 
decisions made is shown in Table V. MPG was not held constant since 
that would imply a simultaneous manipulation of range and the fuel- 


remaining variable. 
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The most striking phenomena evident in Table V is the ratio 
between predicted enemy kills and predicted friendly kills of over four 
to one. As explained by the senior member of the panel of aviators 
after observing this. phenomena: 


"A pilot would be a fool to go into combat is he didn't think 
the odds were stacked heavily in his favor." 


The inverse relationship evident in this table between range and 
friendly kills was considered by the aviators to be a function of aircraft 
type. The F-4 was considered by them to be more of a weapons platform 
than a classical dog-fighter while the MIG-21, a smaller and more 
maneuverable airplane, is better suited for close-in fighting. The 
shorter range and, consequently, time available to the F-4 weapons 
systems decreases some of this system's advantages over a MIG-21 
resulting in a greater enemy opportunity to attack the friendly forces. 
Interesting conjectures E: be made about the results that might be 
observed if the friendly plane were an F-8, a true dog-fighter...The 
slight increase in enemy kills with decreasing range which is evident 
in Table V was felt by the panel of aviators to be the result of anin- 
creased friendly missile kill probability. 

The pushing of the tactical decision to the "must engage" pole 
with shorter ranges was felt to be an indication that a pilot naturally 
considers danger more imminent as the enemy closes; the shorter ranges 


decreasing the protector's options. 
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TABLE V 


VARIABLE MANIPULATION -AVERAGE 
VALUE PER SITUATION 


Variable Value Tactical Decision Enemy Kills Friendly Kills 
Range 4 TO 1.09 3% 
20 2932. 1205 21:5 
Rules of Missile Free 1.93. nes 7 3 
Engagement Eyeball 205 . 84 AS, 
Number of 2 IS AUS Pl 
Enemy 6 20 1219 .41 
Speed 0 we 1242 72 
0 At 27:3 . 28 
Danger 0 22053 94 25 
0 1.03 ALS .29 

Fuel 

Remaining 1000 ZZ 2379 2: 
200 1286 ED >25 
All Free Mean 1.99 I .26 
oe . 1.01 57 
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The differences displayed in the decisions as rules of engagement 
are varied were also felt by the panel to reflect aircraft type. The F-4 
is at its: best when able to deliver the long range, unsuspected missile 
under missiles free conditions. A 40% drop in predicted enemy kills can 
be seen in Table V as the rule of engagement assumes the eyeball con- 
dition. Waiting for ranges to shorten for eyeball identification means 
losing the advantage of the long range punch. The decreased range 
forced on the pilot shows in Table V, as before; in an increase in pre- 
dicted friendly kills but not, surprisingly, in a tactical evaluation pushed 
towards the "must engage" pole. 

The greatly increased number of predicted friendly kills when the 
number of enemy is high, as shown in Table V, is considered to be a 
direct reflection of superiority in aircraft numbers. No matter how 
effective a protector may be, once the protectors are saturated, some 
of the enemy will reach the vulnerable friendly forces. Also, the more 
enemy present the more enemy the friendlies may destroy. The tactical 
decision's migration towards the "not engage" pole was felt by the panel 
to be a sign of caution by the friendly in light of a large enemy force; 
the gains of engagement being more than offset by the possible friendly 
losses. 

The speed differences between the friendly and enemy forces 
appear to push the primary tactical decision towards the two poles of 


engage or not engage. A large speed differential coupled with opening 
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ranges implies, according to the panel, a friendly decision of leaving 

the friendly forces exposed for a long chase or staying with the friendlies 
and not engaging the enemy.. The high closure rate, however, would 
force the friendly to engage but, as shown in the extremely high predicted 
enemy kills in Table V, would place the enemy in the F-4's most optimal 
firing envelope. 

The enemy pointing at the friendly forces places the friendlies in 
the enemy's best firing envelope and, as shown in Table V, seem to 
force the friendly tactical decision to the defensive "must engage" pole. 
As a result, a rise in both predicted enemy and friendly kills can be 
seen. The enemy pointing away from the friendlies with its implied 
opening ranges drives the tactical decision to the "not engage" side 
with concurrent drops in both predicted kills. 

The low enemy kills and "not engage" decision evident with low 
fuel state was interpreted by the panel as a sign for caution for the 
friendly pilot. Friendly kills rise in this low fuel state not only from 
enemy attacks but from destruction due to fuel exhaustion in combat. 
Higher fuel states give the pilot a greater margin in combat for increased 
maneuvering and chasing, resulting in the increased predicted enemy 


kills and greater friendly protection. 


2 





D, MUERTE REGRESSION 

From the multiple regression. analysis a ranking of tonal 
variable importance was made for all pilots and for each rank, hour and 
mission group. Table VI gives these rankings for all pilots and for the 
rank groupings. Appendix G contains the hour and mission rankings. As 
can be seen, MPG or range, one a function of the other, play an important 
role in the tactical decision evaluation. Danger, except in the case of 
Commander where it switches position with speed, is next most important 
followed by speed. 

In the importance of variables for secondary decision evaluation, 
the tactical decision is the most important in evaluating enemy kills 
while the number of enemy is the orly variable more important in evalu- 
ating friendly kills. This may reflect the pilot's integration of his 
primary decision in his next evaluation, i.e., the pilot's "mental set" 
before actually making his next evaluations. 

From this multiple regression analysis predictor equations were 
constructed. Described in Appendix E, the equations were used in 
predicting the decisions made on the decision survey. The obtained 


multiple correlations are described in Table VII and Appendix H. 
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VARIABLE RANKINGS FOR ALL PILOTS AND 
RNA NO eS DECISIONS REACHED 


A. Tactical Decision 


MPG 

Danger 

Speed 

Range 

Number of Enemy 
Rules oí Engagement 
Fuel Remaining 


B: Enemy Kills 


Tactical Decision 
Fuel Remaining 
Rules cf Engagement 
Danger 

Speed 

Range 

Number oí Enemy 
MPG 


E Friendly Kills 


Number oí Enemy 
Tactical Decision 
Speed 

Rules of Engagement 
Fuel Remaining 
Danger 

Range 

MPG 
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TABLE VII 


CORRELATION OF PREDICTED VALUES 
FOR DECISIONS AND ACTUAL DECISIONS 
MADE BY ALL PILOTS AND RANK GROUPINGS 





ALL CDR ICD: ET LT (jg) 
Tactical Decision 20032 200294 262306 OLOT OO 
Enemy Killed A "5 59 .D054 16524 NES 
Friendly Killed ENS 4591 33299 19 7195 MOD 


It appears that the equations are fairly successful for the groups studied 
in predicting their tactical and enemy kill decisions. 

In an attempt to secure a standard of comparison for these equations, 
the panel of aviators was used to predict one of the biographical group's 
decisions. The members of the panel, who were associated with the 
data at some length, considered themselves to be closest to the 
Lieutenant Commander group in rank, hours, and missions. All of the 
Lieutenants on the panel were in the current Lieutenant Commander 
selection zone. They attempted to predict the Lieutenant Commander 
group's decision rather than only making their own evaluations of the 
Situations. The following correlations between the panel's predictions 


and the actual decisions made by the Lieutenant Commander group were 


Found: 
Tactical Decision 29963 
Enemy Killed 5200 
Friendly Killed 4327 
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In essence, the predictor equations fared better than the predicting 


panel of aviators (See Table VII). 
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NAS PROCEDORESARTIFICIAL INTELLIGENCE APPLICATION 





One writer [11] has described and defined artificial intelligence in 


the following way: 


"A human being can think, learn, and create because the program 
his biological endowment gives him, together with the changes in 
that program produced by interaction with his environment after 
birth, enables him to think, learn, and create. If a program thinks, 
learns, and creates, it will be by virtue of a program that endows 
it with these capacities. Clearly this will not be a program - any 
more than the human's is - that calls for highly stereotyped and 
repetitive behavior independent of the stimuli coming from the 
environment and the task to be completed. It will be a program 

that makes the system's behavior highly conditional on the task 
environment - on the task goals and on the clues extracted from 

the environment that indicate whether progress is being made towards 
those goals. It will be a program that analyzes, by some means, 
its own performance, diagnoses its failures, and makes changes 
that enhance its future effectiveness.” 


With the goal of the last sentence above in mind, the panel of 
aviators, all of whom have had student aviator instructor billets, 
described a student pilot's behavior in interaction with his instructor. 
Once past the basic ground schools, flight training appeared to entail 
corrective instruction of the student's cockpit actions. The corrective 
instruction can be broken down into three general areas: 

(1) The student did the opposite of what he should have done. 

(2) The student did too much of what he should have done. 

(3) The student did not do enough of what he should have done. 
Por example; (1) pushing the stick to the left vice the right, (2) applying 


too much throttle, and (3) not using enough flaps. Such mistakes as 
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forgetting to lower the landing. gear were considered as combinations or 
extremes of the three. 

The student's response to the three corrections were: 

(1) Reversing the sign or direction of what was incorrectly done. 

(2) Lessening the magnitude of what was incorrectly done. 

(3) Increasing the magnitude of what was incorrectly done. 
Also, as the student pilot gains more experience his confidence level in 
himself and aircraft rises. His corrections grow smaller in magnitude 
resulting in a smoother development. As stated by one of the aviators 
on the panel: 


"Any movement which is not smooth and gentle in an aircraft is 
probably an incorrect movement." 


In advanced Naval pilot training where such topics as tactics and 
gunnery are taught, the three instructive areas are thought to still exist. 
However, at this level the pilot is developing his evaluative skills 
rather than his motor skills. Since multi-level decisions are involved, 
the student is not allowed to proceed to the next level until the present 
level decision or action is correct. Figure 3 displays this macro-level 
education concept. 

This concept, if captured in a program, would enable the program 
to analyze its performance and diagnose its failures (instructor correction) 
and make changes to enhance its future effectiveness (student modifi- 


cation). This program could show some attributes (confidence and 
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b cunc > 
I e OPE tLOL EDUCATION 


Present situation to student. 
Student evaluates situation and makes primary decision. 
Is decision correct? 

Student gains confidence and experience. 
otudent makes secondary decisions. 


Student modifies own evaluation techniques. 
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experience, learning, fallibility) that, if witnessed in a living organism, 
would certainly be termed intelligent. Hence, an artificial intelligence 
approach to this macro-level pilot learning concept was initiated. 

Certain restrictions were placed on this artificial intelligence 
programming attempt. Since possible graphics adaptations were en- 
visaged, memory size limitations were desired. The available graphics 
terminal at Naval Postgraduate School is an AGT-10 with limited memory. 
Also, as brought out by the panel of aviators, a pilot does not consciously 
search through his memory looking for a situation or solution but rather 
appears to act at a level of perceptual decision making [7]. Likewise, 
the pairel considered the classical tree searching and minimaxing pro- 
cedures as unrealistic for this problem. Computer run time was to be 
as short as possible for both the practical consideration of program turn 
around time and because of the desire for instantaneous decisions as in 
high speed air-to-air combat. Consequently, a polynomial evaluation 
Poth adaptation Of the coefficients technique was chosen along the lines 
of Samuel's checker player [2]. The coefficient would, in essence, act 
as a quasi-memory, reflecting change but not, necessarily, history. 
FORTRAN IV was chosen for the program language over ALGOL and LISP 
because of future graphics applications where no ALGOL or LISP compilers 


would be available. 
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The "instructor" would be one of the pilot groups for which a 
predictive equation existed. The instructor's memory would contain 
the predictive equation and all of the decisions made concerning the 
64 situations presented in the second survey. The "student" would 
contain a predictive equation, either from actual data analysis, or a 


dummy one. It would have no decision history memory. 


A. PROGRAM DESCRIPTION 
The following matrices and vectors of interest are used throughout 
the program: 


(1) sJ (15) i=l, 64;j=1, 6 
Isa ris contalas he six siatlonal variables of the 
second array: 


= 1 - fuel remaining, 

2 — range, 

= 3 - bearing, 

— 4 - heading, 

= 5 - number of enemy, 

6 - rules of engagement. 


ll 


te le ge ge Ne Vs 


They are presented in the 64 different situations, i going 
oa io 62, 


(2) mem (i,j) i=1, 64; j=1, 3 
This matrix contains the three decisions made by the 
Iinstrlelor over che 64 Situations: 


= 1 - primary tactical decision, 
2 — secondary enemy kills decision, 
— 9 — secondary friendly kills decision. 


j 
j 
j 
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prico (1) i=l, 8 

This vector contains the instructor's primary tactical 
decision predictive equation coefficients for the variables 
below: 


— a constant 

- fuel remaining 
range 

number of enemy 
rules of engagement 
— danger 

- Speed 

— MPG 


lI 


ee mr poe xp He ode e po 
I 

On G> CG SF WN Ye 
l 


ll 


startp (i) i=1, 8 
Completely analogous to prico, this vector contains the 
student's primary tactical decision coefficients. 


secoen (i) i=1,9 

This vector contains the instructor's secondary enemy kills 
decision predictive equation coefficients for the same 
variables as in prico. When i= 9, the coefficient is for the 
primary tactical decision reached. 


secofr (i) i=], 9 
This vector is the same as secoen but for the secondary 
friendly kills decision predictive equation. 


starse (i) i=1,9 
Completely analogous to secoen, this vector contains the 
Student's secondary enemy kills decision coefficients. 


Share) IS 
Completely analogous to secofr, this vector contains the 
student's secondary friendly kills decision coefficients. 


kor (i) i=1,8 
inn s io mn F Io l oio [or the instructor's correction of 
the student's startp coefficients. 


kore (i) i=1,9 


are cio olho senicle for the instrúctor's correction of 
ee Aene S Starse Coefficients. 
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(11) korf (i) i=1,9 
Mimceveclorels he vehicle for the instructor S correction of 
merade e Staro cocilicients. 


LE Main Program 
The main program contains, in essence, the previously 

described macro-level pilot education concept. The 64 situations are 
read in (sit(i,j)) followed by the instructor's decisions over these sit- 
uations (mem(i,j)) followed by the instructor's predictive equations 
ico), secoen(i), secoir(i)). This constitutes the instructor's memory 
and expertise, his references and standards. Next, the student is 
brought into the scene by the reading of his starting evaluation equations 
ier) starse (1), Starsil)), these data representing the starting 
student at the beginning of the interactive instructor-student-situational 
process. 

Next, for each of 900 trials, one of the 64 situations is picked 
at random and presented to the student. After calculating danger, speed 
and MPG; a skill hopefully developed through previous training, but 
propably never applied, the student makes the primary tactical decision 
for situation I: 
mE Siamo Stil) 
A stamp) x sit(t,5) 


x sit(1,6) + startp (6) x danger 
x speed + startp (8) x MPG. 


IDECIS = startp (1 
+ startp (3 
u co III 
t startp (7 


Weh Neh ue ee” 


A subroutine INSTRU is now called. This is equivalent to the 


instructor grading the student by saying "You've acted in the wrong 
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ae on nn foomnicn + too little”; or “correct.” Trom this grade the 
student is either sent on to make his secondary decisions (correct) or 
his primary evaluation equation is sent to KORECI subroutine for modi- 
Medion. it the student s gradeis "not correct," upon returning from 
modification he returns to the beginning to have another situation 
presented without attempting to make a secondary decision. If no modi- 
fication is needed, the student's experience and confidence level is 


increased and the secondary decisions are made for situation I: 


MESS tarse (IES AStarse (2) x sit(I, 1) 
taa sida et starse (4) x sit(I,5) 
Tectarses(o) x= sitll 6) * starse (6) x danger 
+ starse (7) x speed + starse (8) x MPG 
Stars OMS IDESIS (enemy kills), 

l es s oI] J .5 + starst (2) x sit(1, 1) 
et Un le SU 2ierestarst (4) x sit(1,5) 
Sao si O) starsí (6) x danger 
+ starsí (7) x speed + starsí (8) x MPG 
Stars) os DECIS (frendiy kills). 


As in INSTRU, these decisions are graded by subroutine GRADER. 
These grades are considered after which either the student's equations 
are modified by KOREII or KORFII or his experience and confidence levels 
are increased. In either case, the student is sent back to the beginning 
for a new situation. The grading and correcting of the secondary enemy 
kill and friendly kill decisions are handled independently, hence 
correction may take place in one but experience and confidence may rise 


the other. 
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Ze Subroutine INSTRU 
INSTRU first determines if it agrees with the student's 
primary tactical decision by checking the instructor's memory for the 
same situation (mem(1,1)). If the instructor and the student agree this 
fact is immediately returned to the main program. lí there is no agree- 
ment, the instructor undertakes a coelficient by coefficient comparison 
of its own live equations (prico) with the student's evaluation 
equation (startp) for the primary decision. First compared as to sign 
and then as to magnitude, a grade is sent back to the main program for 
each coefficient; e.g., wrong sign, too small, too large, or, agrees 
with instructor's coefficient. The coefficient comparison is undertaken 
only when the student's primary decision does not agree with the 
instructor's. 
On subroutine GRADER 
Subroutine GRADER is essentially the same as subroutine 
INSTRU but deals with the grading and coefficient comparison of the two 


secondary decision equations. The two secondary decisions are handled 


independently. 
die oubroutine KORECI 


Subroutine KORECI handles the modification of the student's 
equation for his primary tactical decision. A student's attempted cor- 


rection of the same error is not necessarily stable in magnitude, 
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especially when he is first beginning to learn to fly. As a student's 
experience and confidence grows, however, the modifications become 
more consistent and decreasing in magnitude. Consequently, an 
experience modification factor, 44, is set. Previous runs had shown 
that multiplicative modifiers differing more than about .11 from 1 caused 
an erratic and sudden convergence of the student's coefficient to that 

of the instructor's. Therefore, it was decided to limit the modification 
factor to the closed interval [9, 1.1]. 

The experience modification factor is set by taking a uniformly 
G@ictributed random number from the open interval (0,1). A different 
number is chosen each time KORECI is called. This number is then 
divided by the sum 10 + .1x experience and confidence level. This 
experience and confidence level (X) is simply the number of correct 
primary tactical decisions that have been made. Hence the experience 
modification factor (44) is in the open interval (0,.1). Adding or sub- 
tracting 44 from one produces a modification factor in the open interval 
ol.) that approaches unity. For example; a starting random number 
of .5 with no experience and confidence gives a modification factor of 
RS or 1.05, a starting random number of .5 with an experience and 
confidence level of 200 gives a modification factor of .98 or 1.02, and 
a Starting random number of .9 with an las and confidence level 


oí 1000 gives a modification factor of .99 or 1.01. 
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After the experience modification factor is set, the student's: 
coefficients are checked to see if the variable paired with that co- 
Piieient is even being considered, i.e., is the coefficient zero? If 
so, subroutine NOVICE is called, which means that more intensive 
instruction is given, or, a return is made to a more elementary student 
state. If not, the coefficients' gradings are examined and the co- 
efficients are modified according to the results of these gradings. If 
a wrong sign is indicated, the student's coefficient is multiplied by -1. 
If the coefficient is too great in magnitude, the student's coefficient 
is multiplied by the modification factor 1-44. If the coefficient is too 
small, the modification factor is 1+44. A coefficient determined by 
INSTRU as being correct is not modified. 

Sa Subroutine KOREII 

Subroutine KOREI is essentially the same as subroutine 
KORECI but handles the secondary enemy kill decisions. The experience 
and confidence level is the total number of correct secondary enemy kill 
decisions made by the student. 

6. Subroutine KORFII 

oubroutine KORFII is identical to subroutine KOREII but 
handles the secondary friendly kill decisions. The experience and con- 
fidence level is the total number of correct secondary friendly kill 


decisions made by the student. 
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Te Subroutine NOVICE 
The object of this subroutine is to instruct the student to 
consider a variable that, up to this time, was not being considered 
(coefficient of zero). Since data had revealed that out of 330 coefficients 
considered in the analysis of all pilots and rank, hours, and mission 
groups, 329 fell into the interval (-30,30), a uniformly distributed 
random number is selected from the interval (-30,30) each time NOVICE 
is called. This is then given the sign of the instructor's coefficient 
for that variable and assigned to the student's coefficient in lieu of the 
previous zero value. 
Si Subroutine RANDOM 
This subroutine, a slightly modified version of IBM's RANDU 
[121, simply generates a uniformly distributed random number in the 
interval (0,1). 
(All routines can be found in the Computer Printout Section 


@i this thesis). 


B. RUNNING THE PROGRAM 

The program was run with the average pilot for the entire sample, 
the statistically average pilot from each rank group, the statistically 
average pilot from each hour group and the statistically average pilot 
from each mission group as instructors with the same groups' predictor 


equations as starting coefficients for the students. Therefore, each 
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instructor was run with all oí these average pilots as students. As can 
be seen in Figures 4 through 15 and Table VIII, little differences exist 
between the instructor-student performances. 

As a result of these runs, a synthetic beginning student was con- 
structed with all starting coefficients set to zero. Using this beginner 
model as a student, the program was rerun with: 1) the average pilot 
Mom the entire sample, 2) the Commander group, 3) the 4000-plus hour 
group, and 4) the 300-plus mission group as instructors. The latter 
three groups were chosen for their expertise under the assumption that 
a pilot who has attained higher rank, flown 4000 non-fatal hours or 
survived 300 missions must possess some expertise in flight. The 
results obtained from this “beginner” student run were plotted and 
compared to those of the average student (See Figures 4-15, 16-27 and 
Table VIII). These comparisons and plots were then presented to the 


panel of aviators for their comments and criticisms. 
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TABLE VIII 


RESULTS OF PROGRAM RUNS WITH TWO STUDENTS 


AND FOUR INSTRUCTORS 


Average Pilot as Student: 


line tro Mare First Correci Decision 


4000+ 
5 
5 
= 


4000+ 
40 
8 


300+ 
5 
> 
3 


300+ 
96 
13 


Instructor: Average Pilot DS 
Primary Decision 5 S 
Secondary Enemy Kills 5 5 
Friendly Kills 5 5 
Time* to Attain 90% of Final Correct Percentage After First Correct 
Decision 
lmStructor: Average Pilot CDR 
Primary Decision 5 62 
Secondary Enemy Kills 19 as. 
Friendly Kills 5 D 


Final Correct Decision Proportions 


as truetor: Average Pilot EIE 

Primary Decision . 95 E 

Secondary Enemy Kills Ud 163 
Friendly Kills ,94 E 


Synthetic Beginner as Student: 


Mime” to Make First Correct Decision 


8 


4000+ 
LER 
.60 
107 


4000+ 
199 
072 
161 


4000+ 
48 
32 


35 


3007 
89 
55 
. 64 


300. 
206 
149 
T 


200 
83 
104 


Instructor: Average Pilot GER 
Primary Decision 200 Zo 
Secondary Enemy Kills 129 194 
Pen hile, 9157 77 
Time* to Attain 90% of Final Correct Percentage After First Correct 
Decision 
Instructor: Average Pilot SDR 
Primary Decision 40 47 
Secondary Enemy Kills | Z 40 
Friendly Kills 56 Al 


Pinal Correct Decision Proportions 





Mestructor: Average Pilot CDE 

Primary Decision . 84 205 

Secondary Enemy Kills a Sod 
Friendly Kills Tos" T 


"o 


4000+ 
.86 
.60 
.60 


2295 


SONG ss 
S0 
299 
oi 


Plememcwii Units of trials for the primary decision and in units of correct 


primary decisions for the secondary decisions. 
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V.. RESULTS OF PROGRAM RUNS 


A. PVERAGEH PILOT AS SIUDENT 

For the case of the average pilot for the entire sample as the 
student, a successful evaluation of the situations occurred early in the 
runs, i.e.., within the first five trials for all instructors. As can be 
seen in Table VIII, the average pilot instructing the average pilot had 
the highest overall average of correct decisions: 85% for the primary 
tactical decision, 72% for the secondary enemy kill decision and 94% 
for the secondary friendly kill decision. Naturally, the student's ending 
coefficients were the same as the student's starting coefficients since 
they are identical to the instructor's. The extremely high percentage of 
secondary friendly kill decisions is puzzling in light of the low corre- 
lation coefficient of .4015 betwcen the predictor equation with actual 
secondary kill decisions made. Varying the random number seed changed 
that percentage negligibly. An explanation may lie in the fact that the 
rounding of the program decision equations to the nearest integer improved 
the accuracy of the prediction with respect to the correlation observed. 
For example, a predicted value of 1.49 friendly kills would be rounded 
to one friendly kill by the artificial intelligence program. Assuming the 
actual friendly kill decision to be one friendly kill, the artificial intel- 
ligence program would show a higher correlation than the straight 


predictor equations. 
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On the assumption that a student, upon reaching 90% of his final 
performance, is near matriculation, the time for his running percentage 
of correct decisions to reach 90% of his final percentage after his first 
correct decision was calculated. The running percentage of correct 
decisions reached 90% of.the final percentage within five trials for the 
primary tactica! decision, five correct primary decisions (a correct 
primary es: corresponds to a trial for the secondary decision) for 
the secondary friendly kill decision and 19 correct primary decisions for 
the secondary enemy kill decision. 

The Commander instructor, as Can be seen in Table VIII, whose 
predictor equations differed the most from the average pilot's, showed 
the lowest final percentage of successful decisions except in the case 
of the secondary friendly kill decision; 67% for the primary tactical 
decision, 63% for the secondary enemy kill decision and 71% for the 
secondary friendly kill decision. The high friendly kill decision per- 
centage may be a reflection of a rank-friendly kill correlation witnessed 
in the data analysis (See Table II). The running percentage of correct 
decisions reached 90% of the final percentage within 62 trials for the 
primary decision, 39 correct primary decisions for the enemy kills and 
9/ correct primary decisions for the friendly kills. These longer learning 
times bes felt to be reflections of the instructor and student coefficient 


diil erences. 
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The 4000-plus hour instructor, as can be seen in Table VIII, 
closely matched the average pilot instructor in final percentage of 
primary tactical decisions with 8776 and neared the Commander instructor 
in final percentage of secondary enemy kill decisions with 60%. The 
4000-plus instructor displayed the lowest secondary friendly kill decisions 
with 62%. The time for running percentages to reach 90% of the final 
percentages for this run were moderately low at 40 trials for the primary 
decision, and eight correct primary decisions for both secondary decisions. 
The low number of primary decisions correct needed to reach 90% of the 
secondary percentages is a reflection of the lower final percentages. 

The 300-plus mission instructor, as shown in Table VIII yielded 
final percentages of 83%, 55% and 64% for the primary and secondary 
enemy and friendly kill decisions, respectively. Running percentages 
reached 90% of final percentages within 96 trials for primary decisions, 
73 correct primary decisions for the secondary enemy kill decision and 
95 for the friendly kill decision. These longer learning times are felt 
to be a result of the modification factor (44) rather than a reflection of 
lower percentages or instructor student differences. Although no such 
run was made, the panel of aviators felt that a changed random number 
seed would show reduced learning times. 

However, aS can be seen on the graphs of total decisions right 


versus total trials or total primary decisions correct (See Figures 4-15), 
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the results are similar and nearly linear for all cases. The panel of 
aviators felt this pattern to be consistent with their own experience, 
i.e., as a result of standardization of procedures and training, different 
instructors teaching similar pilots would have, for the most part, similar 
results. They also considered the student, the average pilot from the 
sample, as more experienced than the actual student encountered in a 
flight syllabus. However, in a refresher or replacement training situation 
where the student is experienced, such rapid attainment of consistent 


results as witnessed in the graphs is felt by the panel to be typical. 


B. SYNTHETIC BEGINNER AS STUDENT 

The synthetic beginner student, with all starting coefficients set 
to zero displayed a radically different behavior than that of the average 
Emor (See Table VIII). At the end of 900 trials, all the running per- 
centages for the last 100 decisions were equal to the final percentages 
witnessed when the average pilot was the student. Inspection of the 
runs revealed the interesting phenomenon of long error periods followed 
by rapid learning to a steady percentage. 

With the average pilot as instructor, the first correct primary 
tactical decision was reached on the 200th trial. The first correct 
secondary enemy kill decision was not reached until after 129 correct 
primary decisions had been made, the first correct secondary friendly 


kill decision was not reached until after 157 correct primary decisions. 
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It took 40 additional trials for the correct primary decision percentage 
to reach 90% of the final percentage. The secondary enemy and friendly 
kill decision percentages took 137 and 56 additional correct primary 
decisions, respectively, to reach 90% of their final percentages. 

With the Commander instructor, the first correct primary decision 
was attained after 221 trials, the first correct secondary enemy kill 
and friendly ions after 194 and 177 correct ER decisions, 
respectively. After 47 additional trials, the correct primary decision 
percentage reached 90% of the A The secondary enemy and friendly 
kill decision percentages reached 90% of the final percentages after 40 
and 111 additional correct primary decisions, respectively. 

With the 4000-plus hour instructor, the first correct primary 
decision was reached after 199 trials, the first correct secondary enemy 
and friendly kill decisions coming after 172 and 161 correct primary 
decisions, respectively. Forty-eight additional trials were needed to 
attain 90% of the final correct primary tactical decision percentage, 32 
and 72 additional correct primary decisions were needed to attain 90% 
of the final correct secondary enemy and friendly decision percentages. 

The 300-plus mission instructor used 206 trials before the first 
correct primary tactical decision was reached, 149 and 177 correct 
primary decisions before the first correct secondary enemy and friendly 
kill decisions, respectively. Eighty-three additional trials were needed 


to attain 90% of the final correct primary tactical decision percentage, 
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104 and 225 additional correct primary decisions to reach 90% of the 

first correct secondary enemy and friendly kill percentages, respectively. 
In comparison with the runs and graphs for the average pilot as 

student, three significant points stand out (See Figures 4-15, 16-27 

and Table VIII): 


(1) The beginner requires a lengthy time to attain even one 
correct decision, no matter what decision is being considered. 


(2) After a correct decision, the beginner in general, needs a 
longer learning time to reach the average pilot's degree of 


sophistication. 


(3) Once this degree of sophistication is reached, the beginner 
and average pilot are nearly identical in behavior. 


In essence, the beginner is, indeed, a beginner. In fact, the 
beginner could even be considered the average pilot three hundred trials 
younger. The panel of aviators considered the beginner as analogous to 
the beginning aviation student: the long, flat, error period being as 
in his ground and basic schooling where the basic ideas and guidance 
are implanted; the learning period as the time when a student's motor 
skills are honed to the point that unencumbered mental integration of tho 
entire flight situation is developed - the transition from flight student 
to a full fledged Naval aviator; and the more constant, steady period 
as the reliable activity of an experienced pilot. 

As can be seen in Table VI, dece ic do appear between 
instruction as in the average pilot student runs. Whether these dis- 
crepancies are significant or not is a matter of conjecture. In light 
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of the data analysis showing little correlation of biographical features 
with decisions reached, this question of significant instructor differences 


must remain unanswered unless further research is conducted. 


en To ARTFICIACINTECLIGENCE QUESTION 

Does this program and its resulting runs fall into the classification 
OÍ artificial intelligence? Taking the program in.its entirety, one finds 
that it does analyze its own performances, does diagnose its failures 
and does make changes that enhance its future effectiveness and per- 
formance. This behavior, if witnessed in a human being, would be 
called intelligent behavior. Therefore, by previously cited criteria 
187 111 this program does, indeed, fall into the classification of 


artificial intelligence. 
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VI.. CONCLUSION 


In summation, the low correlations between the biographical 
factors of rank, hours and combat missions ana the decisions made in 
the evaluation. of an air-to-air combat situation is disconcerting. The 
explanation that the standardization of training and flight procedures 
may account for this lack of relationship seems of doubtful validity 
to the author. The one area showing any hint of a relationship is rank 
uudthe prediction of friendly kills. Still, the coefficient of correlation 
is weak (Sce Table II). 

The multiple correlations between the decisions and situational 
variables are moderately low to moderately high, varying from .3378 to 
10193. The lack of any high tión coefficient values may be due 
to randomness in the primary and secondary decision making among the 
pilots. It may also be due to the failure of the multiple regression to 
capture the decision making policy either due to missing variables or 
a failure to capture a non-linear relationship with a linear function. 
Still, in the case of the average pilot, the equations produced from the 
multiple regression predicted 757 correct primary decisions out of 900, 
948 correct enemy kill decisions out of 757, and 714 correct friendly 
Rill decisions out of 757. This indicates a rather adequate capturing 
of the average pilot's decision policy. The relatively low correlation 


statistics may, in this light, be misleading. 
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On. the average, the variables most influencing a pilot in reaching 
a tactical decision of the types shown in Figures_I and II appear to be 
MPG and danger, the least important being fuel remaining and rules of 
engagement (See Table VI). The fact that fuel remaining is considered 
least important while MPG, a function of fuel remaining, is considered 
most important indicates the necessity of considering MPG as a separate, 
though not necessarily independent, function of range and fuel remaining. 

A study by Rigney and De Bow [6] has shown that 63 Combat Infor- 
mation Center watch officers considered range and enemy course (heading) 
as the most important air raid threat variables. Although the watch 
officer's situation is radically different from that of the pilot, the fact 
that range is an argument of the MPG function and course is an argument 
of the danger function is interesting, especially in light of the fact that 
the watch officer does not consider fuel. 

The most important variables considered in making the secondary 
enemy kill decisions are the tactical decision reached, fuel-remaining, 
and rules of engagement (See Table VI). The least important are MPG 
and the number of enemy. An interesting reversal of importance can be 
witnessed in MPG being considered most important in reaching a tactical 
decision and least important in the secondary enemy kill decision. Also, 
the two least important variables in the tactical decision are second and 


third in importance in predicting this secondary enemy kill decision. 
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Number of enemy, tactical decision and speed seem most important 
in predicting pilots’ friendly kill expectancies. Least important in this 
prediction are MPG and range. Again, in comparison with the tactical 
decision, MPG is seen to switch roles. Also noticeable is the number 
of enemy's rise to most important while, in the enemy kill decision, it is 
the next to least important. The three decisions appear to be, in fact, 
three separate decisions, the first (primary tactical) affecting the others 
(enemy kills and friendly kills) but neither of the latter two affecting 
each other. 

Looking at the pilots individually, and within their biographical 
groups, differences do appear to exist in how they reach their tactical 
decisions. But these differences do not appear to be correlated with 
rank, hours or missions but are, rather, insignificant random pertur- 
bations. Secondary decisions show little cross-group variation (See 
Table VI and Appendix G). 

The artificial intelligence program does appear to show adaptive 
learning behavior when the program is taken in its entirety. The behavior 
of the program can reasonably be termed intelligent. Therefore, one 
could safely say that the variables and factors influencing an air-to-air 
combat evaluation can, for the most part, be quantified and utilized in 
an artificial intelligence application of air-to-air combat decision 


making policies. 
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(2) 


(4) 


EL. ~@RitiCAL EVALUATION OF THE STUDY 


Several pertinent critical comments pertaining to this study are: 


The group of 36 pilots from which data were gathered is too small 
to allow any justifiable generalization to Navy pilots in general. 
Consequently, as a result of this small size, any statistics and 
findings generated may not be representative of:Navy pilots in 
general. 


Granted, no generalizations about all Navy fighter pilots can be 
safely made from these findings. An appreciably larger sample 

is required. Still, the sample did cover 2304 separate evaluations. 
Any generalizations drawn do apply, at least, to the group and 
situations studied. 


The assumption that certain factors, such as rank, primary 
decisions, and rules of engagement, can be quantified is ques- 
tionable, i.e., does a Commander possess five times the amount 
of rank as an Ensign? 


Granted, especially in the area of attempting to quantify a variable 
such as rules of engagement. Still, quantifying rank and placing 
the decisions on a continuum are not necessarily unrealistic. 
Perhaps a better substitute for rank would be lineal number and 

the subjects should be asked to give a number value on a decision 
continuum; the relationships between primary decisions as presented 
not necessarily being linear. 


The situational variables such as bearing, heading, fuel states, 
range and number of enemy are not, in reality, restricted to only 
two States. 


Granted, however, questionnaire size limitations enforces such 
a binary state. Still, a better data base could result from a 
broader representation of the situational variables. 


In light of the low biographical-decision correlation, how can one 
assume that expertise is proportional to a biographical attribute? 


The question of expertise, although interesting, is essentially 
a moot one. The only true criteria of expertise in the situation 
would deal with the results of actual air-to-air combat. Since 
the number of people with this kind of expertise is small and not 
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(5) 


(6) 


(7) 


(8) 


available to the researcher, "experts" were picked by the previously 
described method.. Whether, as a result of learning, the student 
has developed into a better or worse fighter pilot cannot be deter- 
mined is irrelevant. What is relevant is whether or not the student 
approaches the standards of his declared "expert" instructor. 


It appears unrealistic to ask a pilot with no combat experience to 
evaluate a combat situation. 


Although seemingly unrealistic on the surface, the armed forces 

do require a person with no combat experience to make an evaluation 
of an actual combat situation. However, the person normally has 
undergone extensive and realistic simulated combat training. All 

of the 36 pilots studied have undergone such training. An interes- 
ting note is the similarity in decisions between the no combat 

group and Lieutenant Commander group which has an average 

combat total of over 160 missions (See Table VI and Appendix G). 


The artificial intelligence program is simply a model of a situation- 
instructor-student interaction. 


Whether the artificial intelligence program is a model or does not 
detract from the fact that it meets the stated criteria of artificial 
intelligence. 


Certain factors may be used for predictors several times in the 
Same equation, such as MPG, range and fuel, raising the multiple 
Correlation accordingly. 


Granted, but the apparent separation of seemingly related functions 
in the ranking scales (See Table VI and Appendix G) and the exper- 
ience of the panel of aviators does not make this practice appear 
unrealistic. 


The student's confidence level increases with success but does 
not decrease with failure, a situation that may be unrealistic. 


Granted, however, when a decreasing confidence level was imple- 
mented, the modification factor in the beginning stages grew so 
large as 10 cause erratic and rapid convergence to the instructor's 
coefficients. Overcoming this problem, however, would not be 
Gimme it. 
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IT eon TINUATIOGROr THE oT UDY 


More data are desperately needed before any of the results un- - 
covered can be safely generalized to Navy pilots in general. Likewise, 
more situations are needed for presentation to the subjects. A factor 
indicating a consideration of the enemy's. plight must also be sought. 

For example, variables indicative of the friendly airplane pointing at 
the enemy or number of friendlies could be considered. 

The computer graphics field is a likely method to use in future 
data collection and in developing real time interactive artificial intel- 
ligence applications. For example, the cathode ray tube screen could 
easily be configured to Rae a cluster of instruments and radar scope. 
The situation would then be presented on the scope via simulated radar 
images, digital readings on the instrument and appropriate character 
generation. Via random number generation a multitude of situations 
could be presented to the subject. Teletype or light pen input would 
increase the subject's interaction and speed of response, allowing 
several hundred responses in the space of an hour. Real time data 
collection and analysis could produce desired statistics and multiple 
regression equations on demand. In the same manner, by using the 
artificial intelligence program described in this study, the human could 
take over the role of instructor with appropriate modifications for response 


collection and analysis. By not utilizing the modification subroutines, 
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the human could take the place of’ the student, his coefficients being 
evaluated and graded by the instructor, followed by a response such 
as "You are not emphasizing your fuel state enough" or "You are neglecting 


your range consideration." The training, selection and research possi- 
bilities in this area are boundless, limited only by the researcher's 


imagination. Through this kind of work, the fighter pilot may becomo 


less of an enigma, less of a superman; more of a human. 
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APPENDIX A 


QUESTIONNAIRE. 


Relative position (range and bearing) of the 
enemy aircraft to the strike force. 


Type enemy aircraft suspected or known. 
I cannot choose between them 


Composition of enemy force--number of raid (s) 
and number of aircraft in each raid. 


Number of assigned TARCAP aircraft available. 
I cannot. choose between them. 

Range of enemy aircraft to the strike group. 
Speed (absoluto) of the enemy aircraft. 

I cannot choose between them. 


Relative position (range and bearing) of the 
enemy aircraft to the strike group. 


Range of enemy aircraft to the strike group. 
I cannot choose between them. 

Type enemy aircraft suspected or a. 
Capabilities and systems operability of your 
c eirerati (e. q. uelistaie, of 
missiles, etc). 

I cannot choose between them. 

Speed (absolute) of the enemy aircrait. 
Altitude of the enemy aircraft. 


I cannot choose between them. 
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I. 


I. 


2. 


a) 


b) 
e 
a) 


b) 


Anti-aircraft, including surface to air mis= 
sile (SAM), activity in your immediate area. 


Capabilities and systems operability of your 
own. aircraft. (e.g. , fuel state, number of 
missiles, etc.). 

I cannot choose between them. 


Altitude of the enemy aircraft 


Composition of enemy force--number of raid (s) 
and number of aircraft in each raid. 


I cannot choose between them. 

Number of assigned TARCAP aircraft available. 
Bearing of the enemy aircraft from yourself. 

I cannot choose between them. 

Bearing of the enemy aircraft from yourself. 


Anti-aircraft, including surface to air mis- 
sile (SAM), activity in your immediate area. 


I cannot choose between them. 


Relative position (range and bearing) of the 
enemy aircraft to the strike force. 


Rate of closure of enemy aircraft to yourself. 
I cannot choose between them. 
Rate of closure of enemy aircraft to yourself. 


Composition of enemy force--number of raid (s) 
and number of aircraft in each raid. 


I cannot choose between them. 


8% 





上 3。 


14. 


B5. 


BG. 


I. 


o. 


Number. of assigned TARCAP aircraft available. 
Range of enemy aircraft to the strike force. 

I cannot choose between them. 

Speed (absolute) of the enemy aircraft. 


Relative position (range and bearing) of the 
enemy aircraft to the strike force. 


I cannot choose between them. 

Capabilities and systems operability of your 
own aircraft (e.g., fuel state, number of mis- 
Siles reto. 

Range of enemy aircraft to the strike force. 

I cannot choose between them. 

Speed (absolute) of the enemy aircraft. 

Type enemy aircraft suspected or known. 


I cannot choose between them. 


Anti-aircraft, including surface to air mis- 
sile (SAM), activity in your immediate area. 


Altitude of the enemy aircraft. 

I cannot choose between them. 

Capabilities and systems operability of your 
eun aircrat (e.g tno sera O mWpumber of 
misses etc.) 


Altitude of the enemy aircraft. 


I cannot choose between them. 
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ES. 


20. 


Zl. 


Ze. 


23. 


24. 


Type of enemy aircraft suspected or known. 


Anti-aircraft, including surface to air mis- 
sile (SAM), activity in your immediate area. 


I cannot choose between them. 


Relative position (range and bearing) of the 
enemy aircraft to the strike force. 


Composition of enemy forces--number of raid(s) 
and number of aircraft in each raid. 


I cannot choose between them. 

Rate of closure of enemy aircraft to yourself. 
Range of enemy aircraft to the strike force. 

I cannot choose between them. 

Number of assigned TARCAP aircraft available. 


Relative position (range and bearing) of the 
enemy aircraft to the strike force. 


I cannot choose between them. 

Capabilities and systems operability of your 
own aircraft (e.g., fuel state, number of mis- 
es eio k 

Speed (absolute) of the enemy aircraft. 

I cannot choose between them. 

Capabilities and systems operability of your 
own aircraft (e.g., fuel state, number of mis- 
Sle cer etc. 


Number of assigned TARCAP aircraft available. 


I cannot choose between them. 
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20. 


26. 


Zils 


28. 


29. 


B. 


Composition of enemy forces --number of raid(s) 


and number of aircraft in each raid. 

Bearing of the enemy aircraft from yourself. 
I cannot choose between them. 

Type enemy aircraft suspected or known. 
Bearing of the enemy aircraft from yourself. 

I cannot choose between them. 

Rate of closure of enemy aircraft to yourself. 


Anti-aircraft, including surface to air mis- 
sile (SAM), activity in your immediate area. 


I cannot choose between them. 


Anti-aircraft, including surface to air mis- 
sile (SAM), activity in your immediate area. 


Range of enemy aircraft to the strike force. 
I cannot choose between them. 
Altitude of the enemy aircraft. 
Type of enemy aircraft suspected or known. 
I cannot choose between them. 
Altitude of the enemy aircraft. 
Number of assigned TARCAP aircraft available. 


I cannot choose between them. 
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31. 


32. 


5. 


34. 


DD. 


66. 


a) 


b) 


Bearing of the enemy aircraft from yourself.. 
Capabilities and system operability of your 
own aircraft (e.g., fuel state, number of mis- 
siles, etc.). 


I cannot choose between them. 


Composition of enemy force--number of raid (s) 
and number of aircraft in each raid. 


Type of enemy aircraft suspected or known. 
I cannot choose between them. 

Bearing of the enemy aircraft from yourself. 
Rate of closure of enemy aircraft to yourself. 
I cannot choose between them. 


Composition of enemy force--number of raid (s) 
and number of aircraft in each raid. 


Anti-aircraft, including surface to air mis- 
siles (SAM), activity in your immediate area. 


I cannot choose between them. 

Rate of closure of enemy aircraft to yourself. 
Capabilities and system operability of your 
own aircraft (e.g. , fuel state, number of mis- 
Ses secte.) 

I cannot choose between them. 

Number of assigned TARCAP aircraft available. 


Speed (absolute) of the enemy aircraft. 


I cannot choose between them. 
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o7. 


Q8. 


39. 


40. 


al. 


42. 


a) 


Speed (absolute) of the enemy aircraft. 


Anti-aircraft, including surface to air mis-- 
siles (SAM), activity in your immediate area. 


I cannot choose between them. 

Range of enemy aircraft to the strike force. 
Altitude of the enemy aircraft. 

I cannot choose between them. 

Type enemy aircraft suspected or known. 
Number of assigned TARCAP aircraft available. 
I cannot choose between them. 

Altitude of the enemy aircraft. 

Bearing of the enemy aircraft from yourself. 

I cannot choose between them. 


Relative position (range and bearing) of the 
enemy aircraft to the strike force. 


Capabilities and systems operability of your 
own aircraft (e.g., fuel state, number of mis- 
cS Eto) 

I cannot choose between them. 

Speed (absolute) of the enemy aircraft. 


Rate of closure of enemy aircraft to yourself. 


I cannot choose between them. 
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ES 


44. 


45. 


46. 


47. 


48. 


Anti-aircraft, including surface to air mis- 
sile (SAM), activity in your immediate area. 


Number of assigned TARCAP aircraft. available. 
I cannot choose between them. 


Anti-aircraft, including surface to air mis- 
sile: (SAM), activity in your immediate area. 


Relative position (range and bearing) of the 
enemy aircraft to the strike force. 


I cannot choose between them. 

Range of enemy aircraft to the strike force. 
Bearing of the enemy aircraft from yourself. 

I cannot choose between them. 

Rate of closure of enemy aircraft to yourself. 
Altitude of the enemy aircraft. 

I cannot choose between them. 


Capabilities and systems operability of your 
own aircraft (e.g., fuel state, number of mis- 
Silo Seto) 


Composition of enemy force--number of raid (s) 
and number of aircraft in each raid. 


I cannot choose between them. 
Range of enemy aircraft to the strike force. 


Composition of enemy force--number of raid (s) 
and number of aircraft in each raid. 


I cannot choose between them. 
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49. 


30. 


SL. 


92. 


23. 


54. 


S. 


Bearing of the enemy aircraft from yourself. 


Relative position (range and bearing) of the 
enemy aircrait to the strike force. 


I cannot choose between them. 

Bearing of the enemy aircraft from yourself. 
Speed (absolute) of the enemy aircraft.. 

I cannot choose between them. 

Range of enemy aircraft to the strike force. 
Type enemy aircraft suspected or known. 

I cannot choose between them. 

Type enemy aircraft suspected or known. 
Rate of closure of enemy aircraft to yourself. 
I cannot choose between them. 

Altitude of the enemy aircraft. 


Relative position (range and bearing) of the 
enemy aircraft to the strike force. 


TTeannot choose between nem: 


Composition of enemy force--number of raid(s) 
and number of aircraft in each raid. 


Speed (absolute) of the enemy aircraft. 

I cannot choose between them. 

Number of assigned TARCAP aircraft available. 
Rate of closure of enemy aircraft to yourself. 
I cannot choose between them. 
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APPENDIX B 


PILOTS RESPONDING TOTMAN- -OUT QUESTIONNAIRE 


DI Gs) WIGS GS) GO BS [KS RS RS OTIS O N N Si SS SS ps 
C) UIS OWN) EM CS CO CO ^ OX» C. CO LO FS CO CO OD ^d CD CO! d» CO DO = © CO 


ON DOP WD Ln 


Rank Hours Missions 
Commander 4500 70. 
Commander 4000 143 
Commander 3200 240 
Commander 3300 354 
Lieutenant Commander 3450 200 
Lieutenant Commander 3200 300 
Lieutenant Commander 3200 140 
Lieutenant Commander 3000 240 
Lieutenant Commander 2700 72 
Lieutenant Commander 2655 160 
Lieutenant Commander A550 11. 
Lieutenant Commander 2500 T 
Lieutenant Commander 2790 150 
Lieutenant Commander 1600 200 
Lieutenant 2350 80 
Lieutenant 1800 249 
Lieutenant 1800 220 
Lieutenant 750 70 
Lieutenant 1700 154 
Lieutenant 1700 75 
Lieutenant 1700 40 
Lieutenant 1500 200 
Lieutenant 1500 100 
Lieutenant 1400 150 
Lieutenant 1400 s 
Lieutenant 7300 90 
Lieutenant RES 100 
Lieutenant 1100 ESO 
Lieutenant 1100 Ro 
Lieutenant TOSO 0 
Lieutenant 1000 150 
Lieutenant 850 0 
Lieutenant 750 0 
Lieutenant 750: 0 
Lieutenant ou 0 
Lieutenant (junior grade) 450 0 
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APPENDIX C 
PANEL OF AVIATORS 


Commander (selectee) - pilot, attack and fighter type aircraft experience: 
two instructor tours in the basic training command; one 
instructor tour with an attack Replacement Air Group; three 
combat tours in Southeast Asia; candidate for M.S. in 
Computer Science. 


Lieutenant Commander - pilot; patrol type aircraft experionce; one 
instructor tour in the basic training command; one combat 
tour in Southeast Asia; candidate for M.S. in Computer 
ocience. 


Lieutenant - pilot; attack type aircraft experience; one instructor tour 
in the Replacement Air Group; one combat tour in Southeast 
Asia; candidate for M.S. in Computer Science. 


Lieutenant - Naval Flight Officer; experience as airborne intercept 
controller; one instructor tour in the basic training command 
teaching computer fundamentals; one combat tour in South- 
east Asia; candidate for M.S. in Computer Science. 


Lieutenant - Naval Flight Officer; experience as airborne intercept 
controller; one tour working on airborne computer display 
systems; one combat tour in Southeast Asia; candidate for 
M.S. in Computer Science. 
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APPENDIX D 


DISCRIMINANT ANALYSIS FOR GROUPS DIFFERENTIATED 
BY BIOGRAPHICAL DATA 


Significance Level of ..05 
Grouped by Rank: 


Chi square = 14.0 
Degrees of Freedom = 21 


No significant differences between groups differentiated 
E 


Grouped by Hours: 


Chi square = 24.0 
Degrees of Freedom = 28 


No significant difference between groups differentiated by 
flight hours. 


Grouped by Missions: 
Chi square = 44.0 


Degrees of Freedom = 28 


No significant difference between groups differentiated 
by combat missions. 
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APPENDIX E 


SETE CTEDPREDIO TORES ON -SOEFRICTE NS 


Primary Tactical Decision --Corresponds to Prico Vector: 


Average Pilot 


SAS 
—.00014 
SU 
USUS 
D 700 
0.0952 
00028 
8.24161 


coc Oc G Hm CQ P3 r4 


CDR 


1225967 
zZ. 00005 
205223 
-03086 
= 07430 
00225 
200025 
Soo VIO 


4. 


4000+ 300+ 
„11902 1.12484 
"0001 3 > OS 
205590 .05204 
05932 .03043 
08141 =. 06924 
.00152 .00440 
.00016 . 00075 
1.1822 22020 


Secondary Enemy Kill Decision - Corresponds to Secoen Vector: 


Average Pilot 


22003 
.00034 
.02213 
.03087 
-11223 
SUZ OL 
0 
2 y 
-.43846 


Oon ana 8 CO [9 = 


CDR 


120750 
.00018 
105597 
. 12057 
.20282 

II 

= can 

SAS 
= OO soe 


27 


4000+ 300+ 
-233/6 1.06406 
00038 .00024 
.05474 FOLTA 
.01666 18579 
294 197 221952 
.00242 ES 
0001, 00008 
20380 lie Pag 
220833 sp 


Secondary Friendly Kill Decision - Corresponds to Secofr Vector: 


Average Pilot 


30915 
00004 
- (0558 
.08816 
209740 
20059 
200013 
De 52823 
— 24 01 


O O jJ G> Ch & DP 
l 


CDR 


233/00 
00012 
—. 01702 
-T8036 
cu 3997 
200079 
.0001° 
2890692 
22010 
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25 


4000- 3001 
.74056 07621 
.00009 200007 
.01714 SUUS 
00989 .30649 
"293,72 2599 
-00025 100278 
.00014 .00024 
. 10520 24.35280 
- 930609 = 28021 





PR po T 
MEAN AND STANDARD DEVIATIONS OF DECISIONS MADE | 


Rank Group Primary Decision Enemy Kills Friendly Kills 
CDR X Ens 1.40 E 
SD .82 US 1.01 
LCDR X 1.96 ‚93 E 
SD .80 .99 ‚64 
LT X 229 DS .18 
SD Ts 1.00 .48 
[IG x E .70 .08 
SD 76 UE Y 
Hours Group 
4000- X Dee ‚83 ‚al 
SD .80 . 78 十 5 
4000 X mes 1.44 aol 
SD ‚79 Be 295 
3000 X 1.92 ‚96 OG 
SD 79 ‚94 an 
2000 X 2.07 1.04 dius 
SD E NUS ‚45 
1000 X 1.99 1.00 E 
SD B . 84 HIE 


Mission Group 


300 x 1.74 o .88 
SD 92 1.02 1.28 
300 X 2.02 NO .19 
SD .74 1.08 .53 
200 X 2.01 ss .23 
SD .80 1.01 EC 
100 X 2.01 RO .23 
SD ‚80 1.04 Cr 
0 X 1.97 1.07 .24 
SD E ‚85 52 
X — arithmetic mean 
D = standard deviation 
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APPENDIX G 


VARIABLE RANKINGS FOR HOUR AND MISSION GROUPS 


HOUR GROUP: 


Primary Tactical Decision 
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Speed 

Range 

Number of Enemy 

Rules of 
Engagement 

Fuel Remaining 
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Secondary Friendly Kill Decision 
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APPENDIX G (continued) 


VARIABLE RANKINGS FOR HOUR AND MISSION GROUPS 


MISSION GROUP: 


Primary Tactical Decision 
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Speed 

Range 

Number of Enemy 

Rules of 
Engagement 

Fuel Remaining 
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Secondary Enemy Kill Decision 
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Danger 

Speed 
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Secondary Friendly Kill Decision 
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APPENDIX H 


JU TIPE C OPREPATONS OPP REDICTOR COUATIONS 
WII TO 


Primary Tactical Secondary Enemy Secondary Friendly 


Decision Kill Decision Kill Decision 
Average Pilot. 5032 10027 ‚4015 
CDR ‚5794 059 . 4391 
OR 0230 29094 25299 
ET OLOT PO SA 
Erg) . 7568 20193 . 9284 
Hours 
4000+ LOO? not 25841) 
3000-4000 20175 29038 .4303 
2000-3000 .6249 59868 5217 
1000-2000 he ls 207/99 23378 
0-1000 "6062 LC DO .4442 
Missions 
300+ BM 27.082 9128 
200-300 .6402 . 5748 .4524 
100-200 one 39915 oo 
1-100 -6309 6077 20309 
0 -6213 6145; 4495 
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